1. Field of the Invention
The present invention relates to an image processing apparatus, a computer-readable recording medium recording an image processing program, and an image processing method for generating a color image with sampling positions where RGB pixels coincide from an RGB Bayer array image.
2. Description of the Related Art
Image pickup apparatuses that pick up optical images to acquire electrical color images are roughly classified into a three-plate image pickup device configuration capable of obtaining three color (for example, RGB three primary colors) signals at a pixel position and a single-plate image pickup device configuration capable of obtaining one of the three color signals at each pixel position.
In general, digital cameras currently available in the market have single-plate image pickup device configurations. Meanwhile, the three-plate image pickup device configuration and the single-plate image pickup device configuration are mixed in digital video cameras currently available in the market.
Constituting the image pickup apparatus with three-plate image pickup device configuration makes it possible to obtain three color signals at each pixel position of a picked up image when picking up an image. Therefore, the image is usually in high quality. However, the image is costly because the structure is complicated requiring more components.
Meanwhile, although the image pickup apparatus with single-plate image pickup device configuration is advantageous in that the structure is simplified compared to the three-plate image pickup configuration, R, G, and B filters need to be arranged in mosaic (see Bayer array as shown in FIG. 5 of the present invention) pixel by pixel to obtain three types of color signals. Therefore, only a single color signal can be obtained at each pixel position of the picked up image. Thus, missing color signals at each pixel position are interpolated using color signals of surrounding pixel positions to obtain three color signals in each pixel in the image pickup device in a single-plate image pickup device configuration with Bayer array.
In the Bayer array, 2×2 pixels is the basic unit of the pixel array. Two pixels of G filters are arranged in one of the diagonal directions in the 2×2 pixels, while one pixel of an R filter and one pixel of a B filter are arranged in the other diagonal direction. Therefore, the sampling density of G pixel is twice as high as the sampling density of R pixel and the sampling density of B pixel. If an image is picked up by such an image pickup device with sample density that differs depending on the color signal, aliasing of a high spatial frequency, which can be expressed in G signals, to the low frequency side occurs in R signals and B signals in an image pickup area with high spatial frequency such as an edge. It is known that the execution of the interpolation process in each color pixel may generate a false color (color that does not usually exist) at the edge or around the edge.
The simplest means to reduce the false color is to reduce the characteristic of a lens or an optical low-pass filter to a spatial frequency that allows reproduction of a sampling interval of R signals or B signals in a Bayer array and form an optical image in the spatial frequency on the image pickup device. However, an image with half the spatial frequency that can be reproduced can be picked up by employing the means, and an unclear image with low resolution is formed. Therefore, in general, an optical system is designed such that the resolution of an optical image formed on the image pickup device shows a spatial frequency characteristic in which the moiré due to the aliasing distortion is not generated at the sampling interval of G signal.
Conventionally, various techniques have been proposed to reduce the false color using an optical system designed in such a manner.
For example, Japanese Patent Application Laid-Open Publication No. 8-237672 describes a technique of applying an interpolation process to set the interpolation positions of three color (R, G, and B) signals to intermediate positions of the pixels both in the horizontal direction and in the vertical direction. The technique focuses on the fact that the interpolation process to the position makes it possible to approximate the frequency characteristics of R, G, and B with half the frequency of the Nyquist frequency. Although the technique described in the Publication is designed to reduce the false color, the false color in an edge area including frequency components close to the Nyquist frequency is not sufficiently suppressed. Furthermore, since the pixels are interpolated at half-pixel positions horizontally and vertically in the technique described in the Publication, attenuation of high frequency components depending on the frequency characteristic of the interpolation filter cannot be prevented, and the resolution is reduced.
Another technique for reducing the false color has been proposed that is different from Japanese Patent Application Laid-Open Publication No. 8-237672 in which the interpolation process of the color signals are realized by filters with uniform characteristics within the image. In the technique, a method is proposed in which an interpolation filter is switched in accordance with at least one direction dependency among a G signal similarity of a local area of the image, a color signal similarity between G signal and R signal, and a color signal similarity between G signal and B signal.
For example, U.S. Pat. No. 5,382,976 describes a technique of calculating a similarity in the horizontal direction and a similarity in the vertical direction among four G signals around a pixel position of a missing G signal (describing with reference to FIGS. 8 to 10 of the present invention, the vertical similarity=|G1−G2|, and the horizontal similarity=|G3−G4|), and the interpolation is performed using only the pixels in the direction with higher similarity. In accordance with the calculated similarity, a G interpolation candidate is selected as an interpolation value of the missing G signal from G interpolation candidates {(G1+G2)/2, (G3+G4)/2, and (G1+G2+G3+G4)/4} obtained by three linear interpolation methods using the G signal as shown in FIGS. 8 to 10 of the present invention.
However, in the technique, an optimal interpolation selection is not performed when the similarities of the G signals in the horizontal direction and the vertical direction are the same as described, for example, in Japanese Patent Application Laid-Open Publication No. 11-275373. Instead, (G1+G2+G3+G4)/4 is selected in some cases. In that case, an interpolation error occurs.
A technique for improving the problem is described in Japanese Patent Application Laid-Open Publication No. 11-275373. Specifically, in the technique described in the Publication, one of vertical interpolation, horizontal interpolation, and interpolation in which the two interpolations are weighted and averaged is selected according to the similarity between an R signal or a B signal at a pixel position for creating an interpolation G signal and surrounding four G pixels (G1, G2, G3, and G4 as shown in FIGS. 8 to 10 of the present invention, and vertical similarity=|X−G1|+|X−G2| and horizontal similarity=|X−G3|+|X−G4| (X is R signal or B signal)). The technique is advantageous in that the false color of an achromatic edge area can be effectively reduced. On the other hand, the chroma may be reduced in an area with high chroma.
The false color is more conspicuously detected in an achromatic edge area. The achromatic area indicates an area in which the values of R signal, G signal, and B signal are substantially the same and the ratio of R signal to G signal as well as the ratio of B signal to G signal are substantially 1. The false color is generated by the ratio of R signal, G signal, and B signal becoming different from 1 (separated) that is supposed to be 1, as a result of the interpolation process.
The technique described in Japanese Patent Application Laid-Open Publication No. 11-275373 is a technique of selecting, from a plurality of prepared interpolation candidates, an interpolation for approximating the ratio of R signal to G signal or the ratio of B signal to G signal to 1. Therefore, generation of false color in the achromatic area is consequently prevented.
However, when the technique is used, an interpolation for approximating the ratio of R signal to G signal or the ratio of B signal to G signal to 1 is also selected in the chromatic area, as in the achromatic area, in which the ratio of R signal to G signal or the ratio of B signal to G signal is basically not 1. Therefore, there is a problem that the chroma that is supposed to be reproduced is reduced in the chromatic area.
The problem will be described with a simple example. In an area considered herein, the color in the horizontal direction is the same, and the color in the vertical direction changes.
It is assumed that the values of R signals are all 200 and the values of G signals are all 10 in a certain line including R signals in a Bayer array. The values are constant values. It is also assumed that the values of G signals located on a line above the focused line are all 5 and the values of G signals located on a line below the focused line are all 45. The values are constant values. B signals on the lines above and below the focused line will not be taken into consideration to simplify the description.
Applying the technique described in Japanese Patent Application Laid-Open Publication No. 11-275373 to the specific value examples, a vertical similarity is |200−5|+|200−45|=350, and a horizontal similarity is |200−10|+|200−10|=380. The vertical similarity is determined to be higher than the horizontal similarity. Therefore, although the horizontal interpolation should be selected, the vertical interpolation is selected. As a result, a G signal interpolation value obtained in the R signal position on the focused line is (5+45)/2=25. In that case, a color difference R−G of the focused line is 200−25=175, which is a value smaller than a color difference R−G=200−10=190 that is supposed to be reproduced. Therefore, the chroma is reduced.
In the conventional technique, the false color may be generated in the edge section, the resolution may be reduced when attempting to reduce the false color, or the image quality may be degraded due to an adaptive error when an adaptive interpolation method using a pixel correlation is applied to prevent the false color or the reduction in resolution when interpolating a missing color signal based on surrounding color signals in a Bayer array image with one color in each pixel that is picked up by a single-plate image pickup device to generate a color image including a plurality of color signals in each pixel. The conventional technique is not enough to prevent the generation of false color, the reduction in resolution, and the reduction in chroma.
The present invention has been made in view of the circumstances, and an object of the present invention is to provide an image processing apparatus, a computer-readable recording medium recording an image processing program, and an image processing method capable of interpolating a missing color signal in an RGB Bayer array image while sufficiently preventing the generation of false color and the reduction in resolution without reducing the chroma.